Accuracy, Agreement, Speed, and Perceived Difficulty of Users’ Relevance Judgments for E-Discovery

This paper presents a study in which four law students and four Library and Information Science (LIS) students judged independently the relevance of documents selected from the e-discovery test collections of the Text REtrieval Conference. The results were compared with the official relevance ground truth and among participants. Given the same task guidelines and minimal training, on average the law assessors achieved the same accuracy on judging relevant documents as the LIS assessors but slightly higher accuracy on judging nonrelevant ones than the latter. Assessors showed moderate to substantial agreements on their relevance judgments. Relevance judgment speed varied markedly among assessors, with a small number of documents costing much more time to review than others. While relevance judgment difficulty is large subjective, all assessors perceived significant distinction between ‘difficult’ judgments and ‘average’ or ‘easy’ ones. Strong correlations were observed between relevance judgment speed and perceived difficulty and between perceived difficulty and relevance judgment accuracy, but not between relevance judgment accuracy and speed. Document subjects, length, and legibility, assessor’s subject knowledge and reading skills, relevance guidelines, and learning effects are identified as factors influencing relevance judgment quality. Implications for building test collections and practicing e-discovery are suggested.